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Abstract 

Over  the  last  few  decades,  services  acquisition  has  continued  to  increase  in  scope  and 
dollars  obligated.  Contracting  for  services  has  grown  in  relation  to  systems  contracting  over 
the  last  couple  of  decades  and  is  the  fastest  growing  procurement  sector  for  the  DoD.  This 
growth  in  dollars  obligated  has  attracted  increased  political  attention  and  scrutiny  on  an 
already  problematic  defense  contracting  process.  The  DoD  has  responded  to  these  problems 
by  improving  services  acquisition  in  several  different  ways,  but  even  with  these 
improvements,  services  acquisition  still  has  problems  in  the  areas  of  procurement  planning, 
source  selection,  and  contract  administration.  This  research  continues  our  ongoing 
investigation  in  DoD  services  acquisition  by  exploring  the  determinants  of  contract  success. 
We  use  the  DoD  Contractor  Performance  Assessment  Reporting  System  (CPARS)  as  a 
proxy  for  contract  success  and  determine  if  there  are  any  relationships  between  contract 
variables  (type  of  service,  contract  dollar  value,  level  of  competition,  contract  type)  and 
contract  success  based  on  CPARS  ratings  (quality  of  product/service,  schedule,  cost  control, 
business  relations,  management  of  key  personnel,  and  utilization  of  small  business).  Our 
research  findings  revealed  that  contract  dollar  value  and  level  of  competition  affected  the 
success  of  a  service  contract.  The  findings  also  revealed  that  the  failure  rate  in  CPARS  was 
lower  than  expected.  Finally,  we  saw  that  as  the  percentage  of  1 1 02  filled  billets  increased, 
the  contract  failure  rate  decreased.  We  also  observed  that  as  workload  dollars  per  filled  billet 
increased,  contractor  performance  ratings  also  increased,  and  thus  contract  failure  ratings 
decreased.  From  these  findings,  we  present  a  discussion  of  the  results  and  the  managerial 
implications. 

Introduction 

The  service  sector  represents  the  largest  and  the  fastest-growing  segment  of  the 
economies  of  the  United  States  and  other  developed  countries.  This  growth  of  services  in 
the  overall  economy  is  also  mirrored  by  the  growth  of  services  acquisition  in  the  Department 
of  Defense  (DoD).  For  example,  DoD  obligations  on  contracts  have  more  than  doubled 
between  fiscal  years  2001  and  2008  to  over  $387  billion,  with  over  $200  billion  spent  just  for 
services  in  2008  (GAO,  2009a).  However,  during  this  time  period,  the  acquisition  workforce 
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declined  from  about  500,000  to  about  200,000  personnel  by  2006  (Gansler,  201 1 ,  p.  237). 
The  downsizing  of  the  defense  acquisition  workforce  has  reduced  the  qualified  contracting 
and  acquisition  workforce  necessary  to  manage  the  increased  service  contract  workload 
(GAO,  2002b,  2009b). 

According  to  the  GAO,  the  poor  management  of  service  contracts  has  undermined 
the  government’s  ability  to  obtain  a  good  value  for  the  money  spent  and  has  contributed  to 
the  GAO’s  decision  to  designate  management  of  services  contracts  as  a  high-risk  area  for 
the  DoD  (GAO,  2013).  In  fact,  as  stressed  in  a  recent  memorandum  for  acquisition 
professionals  from  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics,  improving  the  efficiency  of  acquisition  of  products  and  services  is  of  utmost 
importance  to  the  DoD  (Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  [USD(AT&L)],  2010a).  More  specifically,  in  a  later  memorandum,  the  USD(AT&L) 
focused  on  “improving  tradecraft  in  services  acquisition”  by  strengthening  and  improving  the 
services  contracting  process  (USD[AT&L],  2010b,  p.  5). 

As  the  DoD’s  services  acquisition  continues  to  increase,  the  agency  must  give 
greater  attention  to  proper  acquisition  planning,  adequate  requirements  definition,  sufficient 
price  evaluation,  and  proper  contractor  oversight  (GAO,  2002b).  In  some  ways,  the  issues 
affecting  services  acquisition  are  similar  to  those  affecting  the  acquisition  of  physical 
supplies  and  weapon  systems.  However,  the  unique  characteristics  of  services  and  the 
increasing  importance  of  services  acquisition  offer  a  significant  opportunity  for  conducting 
research  in  the  management  of  services  acquisition  in  the  DoD. 

Research  Methodology 

The  objective  of  this  research  is  to  identify  variables  in  the  services  contracting 
process  that  drive  the  success  of  services  acquisition.  Based  on  the  analysis  of  our  data,  we 
generalize  our  research  findings  and  provide  recommendations  for  improving  the  Army’s  as 
well  as  the  DoD’s  services  acquisition  management.  Our  research  approach  includes 
analyzing  contractor  past  performance  data  obtained  from  the  DoD  Past  Performance 
Information  Retrieval  System  Report  Cards  (PPIRS-RC)  database.  The  PPIRS-RC  database 
consists  of  contractor  performance  assessment  reports  contained  in  the  DoD  Contractor 
Performance  Assessment  Reporting  System  (CPARS).  We  accessed  the  PPIRS-RC 
database  to  collect  contractor  performance  ratings  on  completed  services  contracts 
provided  by  CPARS  to  determine  whether  the  contracts  are  successful  or  not  successful. 
Statistical  analysis  is  used  to  draw  conclusions  on  whether  certain  contracting  variables 
affect  the  success  of  the  contract.  The  specific  contract  variables  analyzed  are  type  of 
service,  contract  dollar  value,  level  of  competition,  and  contract  type.  The  following  are  the 
specific  questions  answered  in  this  research: 

1 .  Do  the  types  of  services  being  acquired  affect  the  success  of  a  service 
contract? 

2.  Do  the  contractual  amounts  affect  the  success  of  a  service  contract? 

3.  Does  the  level  of  competition  used  affect  the  success  of  a  service  contract? 

4.  Does  the  contract  type  affect  the  success  of  a  services  contract? 

Literature  Review 

We  have  addressed  the  need  for  research  in  this  increasingly  important  area  of 
services  acquisition  by  undertaking  six  sponsored  research  projects  over  the  past  six  years. 
The  first  two  research  projects  (Apte,  Ferrer,  Lewis,  &  Rendon,  2006;  Apte  &  Rendon,  2007) 
were  exploratory  in  nature,  aimed  at  understanding  the  types  of  services  being  acquired,  the 
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associated  rates  of  growth  in  services  acquisition,  and  the  major  challenges  and 
opportunities  present  in  the  service  supply  chain. 

The  next  two  research  projects  were  survey-based  empirical  studies  aimed  at 
developing  a  high-level  understanding  of  how  services  acquisition  is  currently  being 
managed  at  a  wide  range  of  Army,  Navy,  and  Air  Force  installations  (Apte,  Apte,  &  Rendon, 
2008,  2009).  The  analysis  of  survey  data  indicated  that  the  current  state  of  services 
acquisition  management  suffers  from  several  deficiencies,  including  deficit  billet  and 
manning  levels  (which  are  further  aggravated  by  insufficient  training  and  the  inexperience  of 
acquisition  personnel)  and  the  lack  of  strong  project-team  and  life-cycle  approaches.  Our 
research  (Apte,  Apte,  &  Rendon,  2010)  also  analyzed  and  compared  the  results  of  the 
primary  data  collected  in  two  previous  empirical  studies  involving  Army,  Navy,  and  Air  Force 
contracting  organizations  so  as  to  develop  a  more  thorough  and  comprehensive 
understanding  of  how  services  acquisition  is  being  managed  within  individual  military 
departments. 

As  a  result  of  these  research  projects  dealing  with  the  service  supply  chain  in  the 
DoD,  we  developed  a  comprehensive,  high-level  understanding  of  services  acquisition  in  the 
DoD,  identified  several  specific  deficiencies,  and  proposed  a  number  of  concrete 
recommendations  for  performance  improvement. 

Based  on  the  foundation  of  the  previously  mentioned  management  theories, 
conclusions  of  the  GAO  and  DoDIG  reports  (Seifert  &  Ermoshkin,  2010),  and  findings  of  our 
own  sponsored  research  projects  on  the  topic,  we  believe  that  the  success  of  services 
acquisition  contracts  is  significantly  influenced  by  four  broadly  defined  factors:  (1)  the  type 
and  quantity  of  services  being  outsourced  and  the  associated  amount  of  acquisition-related 
workload;  (2)  the  characteristics  of  contracts  being  awarded;  (3)  the  capacity  available  to 
carry  out  the  contracting,  project  management,  and  surveillance  work;  and  (4)  various 
management  practices  such  as  use  of  project  team  or  life-cycle  approaches,  and  so  forth. 

The  contract  characteristics  are  affected  by  the  type  of  service  being  acquired,  while 
the  management  practices  being  used  are  influenced  by  the  services  being  acquired,  the 
contract  characteristics,  and,  more  important,  the  capacity  available  to  perform  the 
acquisition  work.  The  success  of  services  contracts,  in  turn,  is  affected  by  the  previously 
mentioned  four  drivers.  This  is  the  fundamental  question  that  motivates  our  in-depth 
research:  What  drives  the  success  of  services  contracts?  This  fundamental  question  is,  of 
course,  critically  important,  and  yet  it  is  also  not  one  that  can  be  answered  easily  or  quickly. 
We  believe  that,  generally,  in  the  case  of  questions  related  to  complex  systems,  it  is 
preferable  to  break  down  the  overall  system  in  smaller  parts,  gain  an  understanding  of  the 
functioning  of  each  part,  and  then  put  all  the  pieces  together  to  better  understand  the  overall 
system  and  answer  the  fundamental  question. 

Based  on  our  conceptual  model,  we  sought  to  understand  how  the  success  of 
services  contracts  is  being  defined  and  measured  by  different  stakeholders.  On  the 
aggregate  level,  our  research  indicated  that,  when  defining  a  successful  service  contract, 
stakeholders  considered  outcomes  (in  the  order  of  performance,  cost,  and  schedule)  slightly 
more  important  than  processes.  Stakeholders  also  ranked  outcome-related  factors  as  most 
important.  Additionally,  on  the  aggregate,  our  research  indicated  that,  when  measuring  a 
successful  service  contract,  stakeholders  considered  outcomes  (in  the  order  of  cost, 
schedule,  and  performance)  more  important  than  processes.  Stakeholders  also  ranked 
outcome-related  factors  as  most  important.  On  the  stakeholder  level,  our  research  indicated 
that,  when  defining  a  successful  service  contract,  PMs,  CORs,  and  COs  considered 
outcomes  (in  the  order  of  performance,  cost,  and  schedule)  slightly  more  important  than 
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processes.  PMs,  CORs,  and  COs  also  ranked  outcome-related  factors  as  most  important. 

On  the  stakeholder  level,  our  research  indicated  that,  when  measuring  a  successful  service 
contract,  PMs,  CORs,  and  COs  considered  outcomes  (in  the  order  of  performance, 
schedule,  and  cost)  more  important  than  processes.  PMs,  CORs,  and  COs  also  ranked 
outcome-related  factors  as  most  important  (Apte  &  Rendon,  2013). 

Building  on  these  research  findings  concerning  how  stakeholders  define  and 
measure  the  success  of  services  contracts,  we  now  explore  the  question  of  what  variables 
in  the  services  contracting  process  drive  the  success  of  services  acquisition.  As  previously 
discussed,  our  research  on  defining  and  measuring  services  contracting  success  found  that, 
on  the  aggregate,  stakeholders  considered  outcomes  slightly  more  important  than 
processes  for  both  defining  and  measuring  success.  Thus,  we  adopt  contract  outcomes, 
specifically  the  outcomes  as  reflected  in  the  contractor  performance  assessment  report,  as  a 
proxy  for  contract  success.  The  next  section  provides  a  brief  background  on  contractor 
performance  information. 

Contractor  Performance  Information 

Contractor  performance  information  is  information  regarding  a  contractor’s 
performance  under  previously  awarded  contracts.  Federal  procurement  policy  requires  that 
agencies  collect  contractor  performance  information  for  contracts  over  $100,000  and  make 
that  information  available  for  use  in  future  contract  award  decisions  (Nash,  Schooner, 
O’Brien-Debakey,  &  Edwards,  2007).  The  collection  of  contractor  performance  information 
occurs  during  the  contract  closeout  phase  using  the  DoD  Contractor  Performance 
Assessment  Reporting  System  (CPARS;  Rendon  &  Snider,  2008).  The  CPARS  assessment 
data  reflect  the  contractor’s  performance  in  specific  areas  including  quality,  schedule,  cost 
control,  business  relations,  management  of  key  personnel,  and  utilization  of  small  business. 
Additionally,  the  CPARS  assessment  rates  the  contractor  in  these  areas  using  the  rating 
scales  Exceptional,  Very  Good,  Satisfactory,  Marginal,  and  Unsatisfactory.  It  should  be 
noted  that  the  contractor  is  allowed  to  review  the  CPAR  assessment  and  provide  comments 
back  to  the  government  assessing  official  prior  to  the  government’s  finalizing  the  CPAR 
report. 

The  contractor  performance  information  reported  in  CPARS  and  accessible  through 
PPIRS  provides  outcome-based  data  that  can  be  used  to  identify  successful  contracts.  The 
successful  contracts  determined  by  using  contractor  performance  information  are  used  in 
our  research  methodology  to  identify  the  contract  variables  that  lead  to  contract  success. 

The  research  methodology  for  this  project  is  discussed  next. 

Research  Design 

With  the  assistance  of  our  MBA  thesis  students  (Hart  et  al. ,  2013),  we  searched  the 
PPIRS  database  to  identify  Army  Mission  Installation  Contracting  Command  (MICC) 
services  (non-systems)  contracts  for  the  period  1996-2013.  This  search  yielded  14,395 
contracts  in  total.  The  data  were  then  refined  to  include  only  those  contracts  associated  with 
the  following  product/service  codes: 

•  R:  Professional,  Administrative,  and  Management  Support  Services 

•  J:  Maintenance,  Repair,  and  Rebuilding  of  Equipment  Services 

•  S:  Utilities  and  Housekeeping  Services 

•  D:  Automatic  Data  Processing  and  Telecommunications  Services 

This  data  refinement  yielded  5,621  contracts.  Our  database  was  further  refined  by 
focusing  on  five  Army  MICC  contracting  organizations.  These  organizations  were  selected 
because  they  are  MICC  field  directorate  offices  (FDOs)  of  this  specific  Army  contracting 
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command.  This  data  refinement  resulted  in  715  service  contracts  that  were  used  in 
conducting  our  analysis,  as  seen  in  Table  1 . 

Table  1.  Database  Breakdown 


(Hartetal.,  2013) 


Total  Contracts 

Total  Army  MICC  Non-System  Contracts 

14,395 

Less:  Non-R,  J,  S,  D  Service  Contracts 

8,774 

Total  R,  J,  S,  D  Service  Contracts 

5,621 

Less:  R,  J,  S,  D  Service  Contracts  at  other 

MICC 

4,906 

R,  J,  S,  D  Service  Contracts  at  MICC  FDO 

Eustis,  Knox,  Hood,  Bragg,  Sam  Houston 

715 

Fort  Eustis 

238 

Fort  Knox 

119 

Fort  Hood 

114 

Fort  Bragg 

55 

Fort  Sam  Houston 

189 

For  each  contract,  data  were  collected  on  specific  contract  variables  (type  of  service, 
contract  dollar  value,  level  of  competition,  contract  type)  and  specific  contractor  assessment 
ratings  (quality  of  product/service,  schedule,  cost  control,  business  relations,  management 
of  key  personnel,  and  utilization  of  small  business).  It  should  be  noted  that  the  data  collected 
from  the  PPIRS  database  were  sanitized  by  removing  identifiable  data  such  as  contract 
number,  contractor  name,  DUNS  number,  and  place  of  performance.  In  addition  to  the 
contractor  performance  information  accessed  from  the  PPIRS-RC  database,  we  also 
collected  MICC  region  organization  demographic  data  (annual  workload  in  dollars,  annual 
workload  in  actions,  number  of  1 1 02  billets  authorized,  and  percentage  of  1 1 02  billets  filled 
(Hart  et  al. ,  2013),  These  data  were  also  analyzed  to  determine  whether  these 
organizational  demographics  were  related  to  contract  success. 

Determining  a  contract  to  be  successful  or  unsuccessful  was  made  based  on 
whether  the  contractor  received  a  marginal  or  unsatisfactory  rating  in  any  of  the  CPAR 
assessment  areas  (quality  of  product/service,  schedule,  cost  control,  business  relations, 
management  of  key  personnel,  or  utilization  of  small  business).  The  contractor’s  receiving  a 
marginal  or  unsatisfactory  rating  in  any  one  of  these  assessment  areas  results  in  the 
determination  of  the  contract  as  unsuccessful. 

Based  on  the  analysis  of  the  data  pertaining  to  contract  variables  and  contractor 
assessment  ratings,  we  answered  our  four  primary  research  questions.  Figure  1  illustrates 
our  research  methodology.  The  column  on  the  right  contains  the  six  CPARS  assessment 
areas.  These  areas  are  used  to  determine  whether  each  contract  is  successful  or 
unsuccessful  based  on  the  CPAR  ratings  (marginal  or  unsatisfactory).  The  successful 
contracts  are  analyzed  using  the  four  contract  variables  shown  on  the  left  column.  The 
purpose  is  to  determine  whether  there  is  a  relationship  between  contract  variables  and 
contract  success. 
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Figure  1.  Research  Methodology 

(Hartet  al.,  2013) 


Findings  and  Analysis 

In  this  section,  we  present  an  analysis  of  our  findings.  As  mentioned  previously,  the 
primary  purpose  of  this  research  is  to  determine  if  there  is  a  relationship  between  contract 
variables  (type  of  service,  contract  dollar  value,  level  of  competition,  contract  type)  and 
contract  success  (based  on  contractor  assessment  rating  of  quality,  schedule,  cost  control, 
business  relations,  management  of  key  personnel,  and  utilization  of  small  business).  With 
the  assistance  of  our  MBA  thesis  students  (Hart  et  al. ,  2013),  we  analyzed  the  data  and 
provide  our  findings  at  the  aggregate  level,  as  well  as  the  contract  variable  level. 

Aggregate  Findings 

The  database  consisted  of  715  contracts  accessed  from  the  PPIRS  database.  Of 
these  contracts,  22  were  determined  to  be  unsuccessful  based  on  the  CPAR  assessment 
area  ratings,  as  described  in  the  previous  section.  This  resulted  in  a  total  contract  failure 
rate  of  3.08%.  These  results  are  seen  clearly  in  Table  2. 

Table  2.  Total  Contract  Information 


(Hart  et  al.,  2013) 


Failures 

Success 

Total 

Failure  Rate 

Contracts 

22 

693 

715 

3.08% 

Additionally,  each  assessment  area  was  given  a  score  associated  with  its  rating 
(Exceptional,  5;  Very  Good,  4;  Satisfactory,  3;  Marginal,  2;  Unsatisfactory,  1).  These 
assessment  area  ratings  were  then  averaged  to  examine  what  assessment  areas  were 
rated  higher  throughout  the  population  (see  Table  3).  As  reflected  in  Table  3,  business 
relations  had  the  highest  average  rate  of  failure  among  the  other  assessment  areas. 
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Table  3.  Average  Success  and  Failure  Rates  of  PPIRS  Areas  of  Contract  Evaluation 

(Hartetal.,  2013) 


Successes 
Average  Ratings 

Failures  Average 
Ratings 

Quality 

4.19 

2.5 

Schedule 

4.19 

2.5 

Cost  Control 

4.1 

2.31 

Business  Relations 

4.17 

3 

Management  of  Key  Personnel 

4.18 

2.68 

Utilization  of  Small  Business 

4.07 

2.5 

The  next  section  briefly  summarizes  our  research  findings  for  each  contract  variable. 
For  further  discussion  of  our  research  findings,  please  reference  out  technical  report  NPS- 
CM-14-001 . 

Type  of  Service:  Finding  1 — The  S  type  services  (Utilities  and  Housekeeping)  had  the 
highest  failure  rate  of  all  the  product  service  codes  analyzed. 

In  answering  the  first  research  question,  “Do  the  types  of  services  being  acquired 
affect  the  success  of  a  service  contract?,”  we  analyzed  the  failure  rates  of  each  service  type. 
The  database  contained  331  R  type  (Professional,  Administrative,  and  Management 
Support)  services,  58  J  type  (Maintenance,  Repair,  and  Rebuilding  of  Equipment)  services, 
292  S  type  (Utilities  and  Housekeeping)  services  and  34  D  type  (Automatic  Data  Processing 
and  Telecommunications)  services. 

The  S  type  services  had  1 1  contract  failures  resulting  in  a  3.77%  failure  rate.  Two 
reasons  tied  for  the  most  common  reasons  for  S  type  service  failures.  These  reasons  were 
six  business  relation  failures  and  six  failures  due  to  management  of  key  personnel.  R  type 
services  had  nine  labeled  failing  contracts  out  of  331 ,  giving  R  type  services  a  failure  rate  of 
2.72%  which  was  the  second  lowest.  The  most  common  reason  for  the  failure  was  quality.  J 
type  services  consisted  of  58  contracts  with  two  labeled  failures.  This  gave  the  J  type 
contracts  a  failure  rate  of  3.45%.  Scheduling  was  listed  as  a  reason  for  both  labeled  failures 
of  the  J  type  services.  There  were  only  34  of  D  service  code  contract  types  with  zero 
failures. 

Although  it  was  found  that  the  type  of  service  was  not  statistically  significant  in  terms 
of  effecting  contract  success,  it  was  observed  that  utilities  and  housekeeping  service 
contracts  failed  six  times  for  business  relations  and  six  times  for  management  of  key 
personnel  out  of  the  1 1  reported  failures.  Mitigating  this  contract  risk  should  focus  on  the 
source  selection  phase,  specifically  the  proposal  evaluation  activity.  During  this  phase  the 
contracting  agency  can  emphasize  its  assessment  of  the  offeror’s  ability  to  coordinate  its 
business  activities,  such  as  its  attitude  towards  customers,  customer  satisfaction,  and 
cooperation.  Additionally,  during  source  selection,  the  contracting  agency  should  increase 
its  emphasis  on  evaluating  the  offeror’s  management  proposal,  and  assessing  the  offeror’s 
ability  to  maintain  qualified  individuals  in  key  positions  as  required  in  the  solicitation. 

Contract  Dollar  Value:  Finding  2 — Contracts  with  a  dollar  value  from  $50  million  to  $1 
billion  had  the  highest  failure  rate  of  all  the  contract  categories. 

In  answering  the  second  research  question,  “Do  the  contractual  amounts  affect  the 
success  of  a  service  contract?,”  we  grouped  the  contracts  into  the  following  categories:  $0 
to  $1  million,  greater  than  $1  million  to  $10  million,  greater  than  $10  million  to  $50  million, 
greater  than  $50  million  to  $1  billion,  and  greater  than  $1  billion. 
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The  $50  million  to  $1  billion  category  consisted  of  92  contracts  with  eight  labeled 
failures,  giving  it  a  failure  rate  of  8.7%.  This  group’s  most  common  reason  for  failing  was 
cost  control,  which  was  listed  for  six  failed  contracts.  This  failure  rate  is  much  higher  than 
the  total  contract  average  failure  rate  of  3.08%. 

In  the  first  group  that  consisted  of  contracts  that  were  worth  $0  to  $1  million,  there 
were  a  total  of  35  contracts.  In  this  first  group,  there  was  only  one  labeled  a  failure.  This 
gave  this  group  a  2.86%  failure  rate.  This  contract  was  labeled  a  failure  because  of  quality. 

The  group  consisting  of  contracts  greater  than  $1  to  $10  million  was  the  largest  of  all 
the  grouped  dollar  amounts.  It  consisted  of  466  contracts,  and  of  those,  10  were  labeled 
failures.  That  gave  this  group  a  2.15%  failure  rate.  While  this  group  had  the  most  failures 
numerically,  it  still  was  under  the  average  failure  rate  because  of  the  number  of  contracts  in 
this  group.  The  most  common  reason  for  this  group  to  fail  was  for  quality,  which  was  cited 
seven  times. 

The  contracts  in  the  greater  than  $10  to  $50  million  group  consisted  of  118  contracts. 
There  were  three  labeled  failures  in  this  group.  This  group  had  a  2.54%  failure  rate.  This 
group  was  also  under  the  average  total  contract  failure  rate  of  3.08%.  This  contract  group 
most  commonly  failed  for  scheduling  issues  and  management  of  key  personnel.  It  failed  for 
scheduling  twice  and  management  of  key  personnel  twice.  This  means  that  one  of  the 
contracts  in  this  group  had  both  issues  listed  as  reasons  for  failure. 

The  group  consisting  of  contracts  worth  greater  than  $1  billion  was  the  smallest 
group  in  our  contractual  amount  grouping.  It  only  consisted  of  four  contracts  and  did  not 
contain  any  labeled  failures. 

Our  analysis  found  that  the  contract  dollar  value  does  have  a  significant  impact  on 
service  contracting  success.  Contracts  with  dollar  values  in  the  $50  million— $1  billion  range 
had  the  highest  failure  rate  at  8.7%.  The  most  common  reason  for  this  failure  was  cost 
control.  Assuming  that  the  awarded  contract  was  proper  (based  on  an  accurate  government 
cost  estimate),  mitigating  this  contract  risk  should  focus  on  the  source  selection  phase. 
During  the  evaluation  of  the  offeror’s  cost  proposal,  the  source  selection  team  should 
accurately  assess  the  offeror’s  ability  to  forecast,  manage,  and  control  the  cost  associated 
with  conducting  the  services.  Additionally,  the  source  selection  team  must  ensure  that  all 
source  selection  evaluators  are  properly  trained  on  how  to  accurately  evaluate  the  offeror’s 
cost  proposal  to  ensure  the  business  fully  understands  the  contract  requirements. 

Our  analysis  also  found  that  the  group  of  contracts  with  values  ranging  from  $1 
million— $1 0  million  had  a  total  of  10  failures,  with  seven  failures  listed  for  quality  reasons. 
Although  not  statistically  significant,  this  reason  for  failure  deserves  further  discussion. 
Mitigating  the  contract  quality  risk  should  focus  on  the  technical  proposal  evaluation  during 
the  source  selection  phase.  The  source  selection  team  should  be  thoroughly  knowledgeable 
about  the  technical  and  quality  requirements  of  the  solicitation.  Only  with  a  competent 
source  selection  team  can  a  sufficient  evaluation  of  the  offeror’s  technical  and  quality 
approach  be  conducted.  A  thorough  evaluation  of  the  offer’s  technical  proposal  will  reveal 
whether  the  offeror  has  the  technical  capability  to  meet  the  contract  quality  requirements. 

Level  of  Competition:  Finding  3 — Contracts  awarded  competitively  had  the  highest 
failure  rate  when  compared  to  the  other  contracts. 

In  answering  the  third  research  question,  “Does  the  level  of  competition  affect  the 
success  of  a  service  contract?,”  we  grouped  the  contracts  into  three  categories:  competitive, 
non-competitive,  and  other.  The  researchers  looked  at  these  different  categories  separately 
and  examined  the  failure  rate  of  each  group. 
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Of  the  540  competitive  contracts,  17  were  labeled  as  failures,  which  yields  a  failure 
rate  of  3.15%.  The  reasons  that  most  often  resulted  in  a  contract  failure  were  in  the  areas  of 
schedule  and  cost  control,  which  were  each  referenced  seven  times.  The  next  highest 
referenced  source  of  failure  was  management  of  key  personnel,  which  was  referenced  six 
times. 


Non-competitive  contracts  had  the  next  highest  failure  rate  at  2.91%.  There  were  172 
non-competitive  contracts  in  the  database,  of  which  five  were  labeled  failures.  Quality  was 
referenced  four  times  while  schedule,  management  of  key  personnel,  and  business  relations 
were  each  referenced  twice. 

Contracts  competed  as  Other  had  three  contracts  in  the  database  with  zero  labeled 
failures. 

Our  research  found  that  competitively  awarded  contracts  had  the  highest  of  the 
category’s  failure  rates  at  3.15%.  These  contracts  failed  seven  times  for  scheduling  reasons 
and  seven  times  for  cost  control.  The  implications  of  these  findings  point  to  the  proposal 
evaluation  aspect  of  the  source  selection  process.  The  source  selection  evaluators  must 
fully  understand  the  cost,  schedule,  and  performance  requirements  of  the  solicitation  in 
order  to  properly  evaluate  the  offeror’s  proposal.  The  offeror’s  proposal  will  reflect  its 
understanding  of  the  contract  requirements  as  well  as  its  capability  for  meeting  those 
requirements.  If  the  government  does  not  do  an  adequate  job  of  evaluating  the  offeror’s 
proposal,  it  increases  the  risk  for  contractor  schedule  delays  and  cost  overruns. 

Contract  Type:  Finding  4 — Contracts  structured  as  a  combination  contract  had  the 
highest  failure  rate  when  compared  to  the  other  five  types  of  available  contracts. 

In  answering  the  fourth  research  question,  “Does  the  contract  type  affect  the  success 
of  a  service  contract?,”  we  grouped  the  contracts  into  six  categories:  Cost  Plus  Award  Fee 
(CPAF),  Combination,  Cost  Plus  Fixed  Fee  (CPFF),  Cost  Plus  Incentive  Fee  (CPIF),  Firm 
Fixed-Price  (FFP),  and  Other.  The  Other  category  includes  all  the  contracts  that  did  not  fit 
into  the  previous  five  categories,  such  as  Labor  Hours  and  Time  and  Materials. 

There  were  four  combination  contracts  examined  in  the  database.  Of  these  four 
contracts,  two  were  labeled  failures,  which  yields  a  failure  rate  of  50.0%.  Schedule  and  cost 
were  both  referenced  twice  in  the  failed  contracts,  while  quality  and  management  of  key 
personnel  were  each  referenced  once. 

Cost  plus  fixed  fee  contracts  had  the  next  highest  failure  rate  at  5.56%.  There  were 
36  CPFF  contracts  in  the  database,  of  which  two  were  labeled  failures.  Cost  control  was 
referenced  twice,  and  schedule  was  referenced  once  in  the  failed  contracts. 

Contracts  competed  as  cost  plus  award  fee  had  58  contracts  in  the  database,  with 
three  of  them  labeled  as  failures.  This  yielded  a  failure  rate  of  5.17%.  Two  of  these  failed 
contracts  referenced  cost  control  and  business  relations,  while  one  referenced  the 
management  of  key  personnel. 

Firm  fixed-price  contracts  had  524  contracts  in  the  database,  with  14  of  them  labeled 
as  failures.  This  yielded  a  failure  rate  of  2.68%.  Seven  of  these  failed  contracts  referenced 
quality,  while  six  referenced  the  management  of  key  personnel. 

Other  contract  types  had  89  contracts  in  the  database  with  one  labeled  as  a  failure 
due  to  quality  and  schedule,  which  yielded  a  failure  rate  of  1.12%.  There  were  four  cost  plus 
incentive  fee  contracts,  which  had  zero  labeled  failures. 

The  type  of  contract  used  in  these  services  acquisition  was  statistically  significant  in 
relation  to  contract  success.  Of  three  categories  of  contract  types  (combination,  CPAF,  and 
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CPIF),  the  combination  type  showed  a  failure  rate  of  50%  (two  failures  out  of  four),  CPAF 
showed  a  failure  rate  of  5.17%,  and  CPFF  a  failure  rate  of  5.56%.  These  last  two  categories 
are  double  the  failure  rate  of  the  FFP  category  which  was  2.67%.  The  leading  and  recurring 
cause  of  contract  failure  for  these  contracts  was  cost  control.  The  implication  of  this  finding 
focuses  us  on  the  procurement  planning  phase  of  services  acquisition.  The  government’s 
effort  during  procurement  planning,  specifically  the  requirements  analysis  and  the  market 
research  activities,  should  ensure  that  the  requirement  and  the  market  are  sufficiently 
analyzed  in  order  to  select  the  appropriate  contract  type.  The  more  defined  the  requirement, 
the  more  appropriate  it  is  to  use  a  fixed-price  contract.  In  the  event  that  a  cost-type  contract 
is  selected,  the  government  should  ensure  that  an  accurate  cost  estimate  is  developed  to  be 
used  in  cost  negotiations.  Additionally,  during  the  source  selection  phase,  specifically  the 
cost  proposal  evaluation  activity,  the  government  should  ensure  that  the  offeror’s  cost 
proposal  accurately  reflects  a  thorough  understanding  of  the  service  effort  and  the  related 
costs.  If  the  government  does  not  do  an  adequate  job  in  evaluating  the  offeror’s  proposal, 
the  risk  increases  for  contractor  schedule  delays  and  cost  overruns.  If  the  government  does 
not  conduct  a  sufficient  evaluation  of  the  offeror’s  cost  proposal,  it  may  award  the  contract  to 
an  offeror  who  does  not  have  an  adequate  understanding  of  the  service  requirement  and 
thus  leads  to  greater  possibility  of  cost  overruns.  In  addition  to  the  contract  variables 
previously  discussed,  we  also  analyzed  other  variables  to  determine  whether  they  had  any 
effect  on  contract  success.  The  other  variables  include  MICC  annual  workload  (by  dollar 
value),  MICC  annual  workload  (by  number  of  actions),  number  of  1 102  billets,  and 
percentage  of  1102  billets  filled.  We  also  analyzed  the  data  in  terms  of  workload  (in  dollars) 
per  filled  billet.  The  next  section  discusses  the  findings  related  to  these  variables. 

Significance  Testing 

We  further  analyzed  our  data  to  determine  whether  any  of  the  variables  had  a 
significant  relationship  with  contract  success  by  specifically  looking  at  the  contract  failure 
rates.  We  used  the  chi-square  test  (Fisher’s  exact  test)  to  test  whether  the  actual  failure 
rates  are  significantly  different  than  from  what  would  be  expected  if  the  total  contract  failure 
rate  was  applied  to  each  variable.  The  null  hypothesis  for  this  test  was  that  the  category 
failure  rates  within  the  variables  are  not  significantly  different  from  the  total  contract  failure 
rate  (3.08%).  We  reject  the  null  hypothesis  if  the  p-value  for  the  variable  is  less  than  .05. 

The  results  of  the  chi-square  test  are  reflected  in  Table  4. 


Table  4.  Chi-Square  and  Fisher’s  Exact  p-Value  Test  Results 

(Hartetal.,  2013) 


Contract  Variables 

p-value 

Significant? 

Type  of  Service  (RJSD) 

0.761 

No 

Contractual  Amounts 

0.036 

Yes 

Level  of  Competition 

1.00 

No 

Contract  Type 

0.009 

Yes 

The  chi-square  test  resulted  in  only  two  variables  being  significantly  different  when 
the  average  total  contract  failure  rate  is  applied  to  the  variable  (3.08%).  These  variables 
were  contract  dollar  value  (p  =  .036)  and  contract  type  (p  =  .009).  In  the  contract  dollar  value 
variable,  the  category  of  $50  million  to  $1  billion  had  an  expected  failure  rate  of  1  but  had  an 
actual  failure  rate  of  8.  In  the  contract  type  variable,  the  category  of  Combination  Type 
contracts  was  expected  to  have  a  0.1  failure  but  actually  had  2  failures.  Additionally,  FFP 
Type  contracts  were  expected  to  have  16  failures  but  actually  had  14  failures.  Finally,  Other 
Type  contracts  were  expected  to  have  3  failures  but  only  had  1  failure. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-510- 


MICC  Workload  in  Dollars:  Finding  5 — Regional  MICC  offices  that  had  spent  between 
$0  and  $500  million  in  annual  workload  had  the  highest  failure  rate. 

We  grouped  the  contracts  into  three  categories  based  on  the  MICC  regional  office’s 
annual  workload  (dollars  obligated).  The  categories  consisted  of  total  dollars  obligated 
between  $0  and  $500  million,  greater  than  $500  million  to  $1  billion,  and  greater  than  $1 
billion. 

The  first  category,  MICC  regions  that  obligated  between  $0  and  $500  million, 
consisted  of  344  contracts  and  had  a  failure  rate  of  4.36%.  The  primary  reason  stated  for 
failure  in  this  category  was  schedule. 

The  second  category,  MICC  regions  that  obligated  greater  than  $500  million  to  $1 
billion,  consisted  of  256  contracts  with  a  failure  rate  of  2.34%.  Management  of  key  personnel 
was  stated  as  the  reason  for  failures. 

The  third  category,  obligations  greater  than  $1  billion,  contained  115  contracts  with 
only  one  of  them  labeled  as  a  failure,  giving  it  a  0.87%  failure  rate.  The  labeled  failure  listed 
cost  control  and  business  relations  as  the  reason  for  failure. 

MICC  Workload  in  Actions:  Finding  6 — MICCs  that  completed  3,501  to  7,000  contract 
actions  annually  had  the  highest  failure  rate  when  compared  to  MICCs  that  completed 
3,500  or  fewer  contract  actions. 

We  grouped  the  contracts  into  two  categories  based  on  the  MICC  regional  office’s 
annual  workload  (actions  completed).  The  first  category  ranged  from  0  to  3,500  contract 
actions.  The  second  category  ranged  from  3,501  to  7,000  contract  actions. 

The  3,501  to  7,000  contract  actions  category  consisted  of  277  contracts.  Of  these 
277  contracts,  nine  were  labeled  as  failures,  which  yielded  a  failure  rate  of  3.25%.  Quality 
and  the  management  of  key  personnel  were  each  referenced  five  times  in  the  failed 
contracts. 

The  0  to  3,500  contract  actions  category  consisted  of  413  contracts  of  which  12  were 
labeled  as  failures,  resulting  in  a  failure  rate  of  2.91%.  Schedule  and  cost  control  were  each 
referenced  five  times  while  quality  was  referenced  four  times  in  the  failed  contracts.  There 
were  no  data  available  for  the  25  contracts  of  which  one  was  labeled  a  failure. 

1102  Billets  Authorized:  Finding  7 — The  category  with  0  to  50  1102  billets  had  the 
highest  failure  rate. 

We  grouped  the  contracts  into  three  categories  based  on  the  number  of  1102  billets 
authorized  for  each  MICC  regional  office.  The  first  category  consisted  of  MICC  regional 
offices  that  had  0  to  50  1102  billets,  the  second  category  consisted  of  MICC  regional  offices 
that  had  over  50  authorized  1102  billets.  (There  were  25  contracts  that  did  not  have  any 
data  available). 

MICC  regional  offices  that  had  0  to  50  authorized  1102  billets  consisted  of  147 
contracts  with  eight  failures,  giving  these  MICC  offices  a  failure  rate  of  5.44%.  The  common 
reason  for  these  failures  was  quality.  This  reason  was  listed  for  five  of  the  eight  labeled 
failed  contracts. 

MICC  regional  offices  that  had  over  50  authorized  1102  billets  consisted  of  543 
contracts  with  13  failures,  giving  this  group  a  2.39%  failure  rate.  The  most  common  reason 
for  failure  listed  for  this  group  was  schedule.  This  reason  was  listed  seven  times  out  of  the 
13  labeled  failed  contracts. 
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The  final  category  did  not  have  data  on  authorized  billets.  This  group  contained  25 
total  contracts  with  one  failure,  giving  this  category  a  4%  failure  rate. 

1102  Billets  (Percentage  Filled):  Finding  8 — As  the  percentage  of  1102  filled  billets 
increased,  the  contract  failure  rate  decreased. 

We  grouped  the  contracts  into  six  categories  based  on  the  percentage  of  1 102  billets 
filled  for  each  MICC  regional  office.  The  categories  consisted  of  MICC  regional  offices  that 
had  billets  50-60%  filled,  61-70%  filled,  71-80%  filled,  81-90%  filled,  and  91-100%  filled. 
(There  were  also  contract  data  that  did  not  contain  adequate  billet  information.) 

The  MICC  region  group  of  1 102  billets  that  were  61-70%  filled  had  81  contracts,  with 
a  4.94%  failure  rate.  The  most  common  reasons  listed  for  this  group  to  fail  were  schedule 
and  cost  control. 

For  the  MICC  region  group  of  1 1 02  billets  that  were  50-60%  filled  had  22  contracts 
with  one  labeled  failure,  which  gave  this  group  a  4.55%  failure  rate.  This  rate  is  higher  than 
the  total  contract  average  of  3.08%.  The  reason  for  the  failure  in  this  group  was  quality, 
scheduling,  and  management  of  key  personnel. 

The  next  group  of  1102  billets  that  were  71-80%  filled  consisted  of  122  total 
contracts  with  five  contracts  labeled  as  failures,  giving  this  group  a  4.1%  failure  rate.  This 
group  was  higher  than  the  total  contract  failure  rate  of  3.08%.  The  most  frequent  reasons 
listed  for  the  contract  failures  were  schedule  and  cost  control. 

The  group  that  contained  billets  filled  81-89%  had  233  contracts  with  five  labeled  as 
failures,  giving  this  group  a  2.15%  failure  rate.  The  most  common  reasons  for  failure  of 
these  contracts  were  quality,  schedule,  and  business  relations. 

The  data  for  the  final  group  that  had  billets  90-100%  filled  consisted  of  99  total 
contracts.  It  contained  one  labeled  failure,  giving  this  group  a  failure  rate  of  1.01%.  The 
reason  for  the  failure  in  this  group  was  management  of  key  personnel. 

Although  it  was  found  that  the  percentage  of  1 102  billets  filled  was  not  statistically 
significant  in  terms  of  effecting  contract  success,  it  was  observed  that  as  the  percentage  of 
1102  filled  billets  increased,  the  contract  failure  rate  decreased.  Thus,  as  the  MICCs 
contracting  workforce  increased,  the  contract  success  rate  increased.  This  finding  supports 
the  previous  reports  on  the  importance  of  having  trained  1102  personnel  (GAO,  2002b, 
2009a;  Gansler,  2011).  Having  a  sufficient  contracting  workforce  enables  the  government  to 
perform  due  diligence  not  only  in  the  source  selection  phase  of  the  contract,  but  also  in  the 
contract  administration  phase  as  well. 

Workload  Per  Filled  Billet:  Finding  9 — As  workload  dollars  per  filled  billet  increases, 
contractor  performance  ratings  also  increase,  and  thus  contract  failure  ratings 
decrease. 

In  addition  to  analyzing  the  data  in  terms  of  workload  (both  dollars  and  actions)  as 
well  as  billets  (both  authorized  and  percentage  filled),  we  also  analyzed  the  data  by  looking 
at  workload  (in  dollars)  per  filled  billet  as  it  correlated  with  associated  failure  rates  and 
average  CPARS  ratings  across  seven  dimensions  for  which  contracts  are  rated.  Our 
purpose  here  was  to  determine  whether  workload  (in  dollars)  per  filled  billet,  as  a  measure 
of  contracting  work  being  performed  by  an  individual,  is  related  to  contract  failure  rate,  as 
reflected  in  performance  outcome. 

We  determined  a  coefficient  of  correlation  between  each  average  performance  and 
workload  per  filled  billet.  A  positive  correlation  indicates  that  both  workload  and  the 
performance  measure  increase  together.  In  addition,  we  created  scatter  plots  to  visually 
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analyze  the  relationship  between  workload  per  filled  billet  and  each  of  the  individual 
performance  measures.  The  results  from  Fort  Eustis  in  2007  were  excluded  from  both  the 
correlation  determination  and  the  scatter  plots  since  the  workload  per  filled  billet  of  $45 
million  was  determined  to  be  an  outlier. 

As  reflected  in  the  table  and  charts,  the  correlations,  although  not  very  strong, 
indicate  that  as  workload  dollars  per  filled  billet  increase,  contractor  performance  ratings 
also  increase,  and  thus  contract  failure  ratings  decrease.  Although  the  correlations  are  not 
particularly  strong,  they  do  provide  some  interesting  preliminary  findings  that  deserve  further 
exploration.  This  relationship  could  be  explained  by  reasoning  that  the  MICC  with  higher 
workload  (dollars  per  filled  billets)  may  not  have  the  resources  to  ensure  due  diligence  in 
conducting  thorough  contractor  performance  assessments  and  thus  may  be  inflating  the 
performance  ratings.  Another  explanation  may  be  that  MICCS  with  higher  workloads  may 
have  more  experienced  and  competent  personnel  and  more  capable  contracting  processes 
that  are  resulting  in  better  contracts  and  successful  contractors.  As  previously  stated, 
although  these  correlations  are  not  particularly  strong,  they  do  provide  some  interesting 
preliminary  findings  that  deserve  further  exploration  in  future  research. 

Summary,  Conclusions,  and  Recommendations 
Summary 

Although  the  service  sector  represents  the  largest  and  the  fastest-growing  segment 
of  the  U.S.  economy  and  comprises  over  half  of  the  DoD’s  procurement  dollars,  poor 
management  practices  have  undermined  the  government’s  ability  to  obtain  best  value  in 
services  acquisition.  The  DoD  must  give  greater  attention  to  proper  procurement  planning, 
source  selection,  and  contract  administration  in  acquiring  services.  The  use  of  contractor 
performance  information  is  a  critical  facet  of  services  acquisition.  During  the  source 
selection  phase  of  the  contract  management  process,  specifically  proposal  evaluation,  the 
DoD  relies  on  the  Past  Performance  Information  Retrieval  System  (PPIRS)  to  obtain  an 
offeror’s  past  performance  information,  a  required  evaluation  factor  in  source  selections. 
Additionally,  during  the  contract  closeout  phase,  specifically  after  the  contract  period  of 
performance,  the  DoD  assesses  the  contractor’s  performance  and  reports  this  assessment 
in  the  Contractor  Performance  Assessment  Reporting  System  (CPARS). 

The  objective  of  this  research  was  to  identify  variables  in  the  services  contracting 
process  that  drive  the  success  of  services  acquisition.  Our  research  approach  included 
analyzing  contractor  past  performance  data  obtained  from  the  DoD  PPIRS  database.  We 
accessed  the  PPIRS  database  to  collect  contractor  performance  ratings  on  completed 
services  contracts  to  determine  whether  the  contracts  are  successful  or  not  successful. 

Using  statistical  analysis  to  draw  conclusions  on  whether  certain  contracting  variables,  type 
of  service,  contract  dollar  value,  level  of  competition,  and  contract  type  affected  the  success 
of  the  contract,  we  generalized  our  research  findings  and  provided  recommendations  for 
improving  the  Army’s  as  well  as  the  DoD’s  services  acquisition  management.  Our  focus  is  to 
identify  where  the  contract  failures  could  be  addressed,  thus  assisting  DoD  contracting 
officials.  In  presenting  the  conclusion  of  our  research,  we  have  to  first  assume  that  the 
awarded  contract  was  proper  (based  on  a  fair  and  reasonable  price  and  accurately  reflecting 
schedule  and  performance  requirements)  in  terms  of  the  requiring  agency’s  needs.  With  a 
properly  awarded  contract,  the  DoD  can  then  perform  the  required  contract  administration 
activities  in  order  to  identify  deficiencies  with  the  contractor’s  work  and  ensure  proper 
documentation  into  CPARS. 
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Conclusion 

The  following  presents  our  research  questions  and  related  findings: 

1 .  Do  the  types  of  services  being  acquired  affect  the  success  of  a  service 
contract? 

The  S  type  services  (Utilities  and  Housekeeping)  had  the  highest  failure  rate 
of  all  the  product  service  codes  analyzed.  The  S  type  services  had  1 1 
contract  failures,  resulting  in  a  3.77%  failure  rate.  The  reasons  for  contract 
failure  included  business  relations  and  management  of  key  personnel. 

2.  Do  the  contractual  amounts  affect  the  success  of  a  service  contract? 
Contracts  with  a  dollar  value  from  $50  million  to  $1  billion  had  the  highest 
failure  rate  of  all  the  contract  categories.  The  contracts  with  dollar  values 
between  $50  million  to  $1  billion  category  consisted  of  92  contracts,  with 
eight  labeled  failures  giving  it  a  failure  rate  of  8.7%.  This  group’s  most 
common  reason  for  failing  was  cost  control. 

3.  Does  the  level  of  competition  used  affect  the  success  of  a  service  contract? 
Contracts  awarded  competitively  had  the  highest  failure  rate  when  compared 
to  the  other  two  forms  of  competition  available.  Of  the  540  competitive 
contracts,  17  were  labeled  as  failures,  which  yield  a  failure  rate  of  3.15%.  The 
reasons  that  most  often  resulted  in  a  contract  failure  were  in  the  areas  of 
schedule  and  cost  control. 

4.  Does  the  contract  type  affect  the  success  of  a  services  contract? 

Contracts  structured  as  a  combination  contract  had  the  highest  failure  rate 
when  compared  to  the  other  five  types  of  available  contracts.  There  were  four 
combination  contracts  examined  in  the  database.  Of  these  four  contracts,  two 
were  labeled  failures,  which  yields  a  failure  rate  of  50.0%.  Schedule  and  cost 
were  both  referenced  twice  in  the  failed  contracts,  while  quality  and 
management  of  key  personnel  were  each  referenced  once. 

We  further  analyzed  our  data  to  determine  whether  any  of  the  variables  had  a 
significant  relationship  with  contract  success  by  specifically  looking  at  the  contract  failure 
rates.  We  used  the  chi-square  test  (Fisher’s  exact  test)  to  test  if  the  actual  failure  rates  are 
significantly  different  than  what  would  be  expected  if  the  total  contract  failure  rate  was 
applied  to  each  variable.  The  results  of  the  chi-square  test  identified  that  Contractual 
Amounts  and  Contract  Type  were  our  only  statistically  significant  variables. 

Finally,  we  looked  at  the  relationships  between  percentage  of  filled  1102  billets  and 
failure  rates,  and  between  workload  dollars  per  filled  billet  and  failure  rates,  and  made  some 
interesting  observations.  We  saw  that  as  the  percentage  of  1 102  filled  billets  increased,  the 
contract  failure  rate  decreased.  This  would  seem  intuitive:  that  as  the  workforce  increases, 
the  contract  success  rate  would  also  increase,  since  there  would  be  sufficient  resources  to 
manage  the  contracting  process.  However,  we  also  observed  that  as  workload  dollars  per 
filled  billet  increased,  contractor  performance  ratings  also  increased,  and  thus  contract 
failure  ratings  decreased.  This  relationship  appears  counterintuitive,  since  any  additional 
workload  in  the  organization  would  place  a  higher  demand  on  the  workforce,  thus  resulting 
in  fewer  resources  to  manage  the  contracting  process.  These  two  observations  need  further 
exploration  using  a  much  expanded  contractor  performance  database. 

Recommendations 

The  overarching  goal  in  our  services  acquisition  research  stream  is  to  identify  the 
drivers  of  acquisition  practices  and  their  relationship  to  successful  service  contracts,  or,  in 
other  words,  “What  drives  successful  services  contracts?”  Our  previous  research  indicated 
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that  the  DoD  defines  and  measures  a  successful  services  contract  by  analyzing  the  contract 
outcomes  (performance,  cost,  and  schedule).  We  used  contractor  performance  as  a 
surrogate  measure  for  contract  outcomes;  that  is,  a  successful  contractor  as  reflected  in  the 
CPAR  (in  terms  of  quality,  schedule,  cost  control,  business  relations,  management  of  key 
personnel,  and  utilization  of  small  business)  would  indicate  a  successful  contract.  Thus,  our 
research  used  contractor  performance  ratings  to  determine  if  there  was  any  relationship 
between  contract  variables  and  contract  success.  Based  on  our  research,  we  provide  the 
following  recommendations  for  our  more  meaningful  research  findings. 

Finding  1  stated  that  the  S  type  services  (Utilities  and  Housekeeping)  had  the 
highest  failure  rate  of  all  of  the  product  service  codes  analyzed.  Although  this  finding  was 
not  statistically  significant,  it  may  indicate  that  contracting  for  utilities  and  housekeeping 
services  may  be  more  difficult  than  for  the  other  service  types.  Perhaps  the  contracting 
agencies  may  need  to  place  additional  emphasis  on  procurement  planning  (developing  the 
utilities  and  housekeeping  services  performance  work  statement  [PWS])  as  well  as  source 
selection  (evaluation  of  offeror  technical  proposals).  Additionally,  increased  emphasis  during 
contract  administration  (contractor  oversight  and  surveillance)  may  also  improve  the 
success  of  utilities  and  housekeeping  services  contracts. 

Finding  2  shows  that  contracts  with  a  dollar  value  from  $50  million  to  $1  billion  had 
the  highest  failure  rate  of  all  contract  dollar  value  categories.  The  most  common  reason  for 
these  contract  failures  was  due  to  cost  control  factors.  This  finding  was  found  to  be 
statistically  significant  and  provides  an  indication  that  perhaps  contracts  in  this  dollar  value 
range  require  additional  management  review  and  oversight  during  the  procurement  planning 
and  source  selection  phases.  The  contracting  agencies  may  not  be  providing  adequate 
review  for  these  higher  value  contracts  in  the  areas  of  developing  PWSs,  cost  estimates, 
budgets,  and  service  delivery  schedules. 

Finding  3  identified  that  contracts  awarded  competitively  had  the  highest  failure  rate 
when  compared  to  contracts  awarded  non-competitively.  The  most  common  reason  for 
these  contracts  failures  was  due  to  schedule  and  cost  control  factors.  Although  this  finding 
was  not  statistically  significant,  it  may  indicate  that  additional  emphasis  may  be  needed  in 
conducting  competitive  source  selections.  Perhaps  the  contracting  agencies  may  need 
additional  training  in  offeror  proposal  evaluation,  specifically  evaluating  cost  and  technical 
proposals,  as  well  as  evaluating  project  schedules.  The  offeror’s  cost  and  technical 
proposals  and  schedule  reflects  the  offeror’s  understanding  of  the  contract  requirements.  If 
these  proposals  and  schedules  are  not  properly  evaluated  by  trained  source  selection  team 
members,  the  contractor  may  experience  cost  overruns  and  schedule  delays. 

Finding  4  identified  that  contracts  structured  as  combination  contracts  had  the 
highest  failure  rate  when  compared  to  the  other  five  types  of  available  contracts  (CPAF, 
CPIF,  CPFF,  FFP,  Other).  This  finding  was  found  to  be  statistically  significant  and  provides 
an  indication  that  perhaps  services  contracts  consisting  of  a  combination  of  contract  types 
are  more  complex  and  require  additional  management  review  and  oversight  during  the 
contracting  process.  The  contracting  agencies  may  not  be  providing  adequate  review  of 
these  more  complex  contracts  in  the  areas  of  developing  PWSs,  cost  estimates,  budgets, 
and  service  delivery  schedules.  In  addition,  the  contracting  agencies  may  need  to  provide 
additional  training  to  the  contracting  workforce,  specifically  in  the  areas  of  dealing  with 
combined  cost  type  and  fixed  price  services  contracts. 

Our  final  recommendation  is  based  on  Finding  8,  which  showed  that  as  the 
percentage  of  1102  filled  billets  increased,  the  contract  failure  rate  decreased.  Although  this 
was  not  found  to  be  statistically  significant,  it  does  show  that  as  the  agencies’  contracting 
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workforce  increased,  the  contract  success  rate  increased  as  well.  Thus,  our 
recommendation  is  that  the  contracting  agencies  strive  to  ensure  that  there  are  sufficient 
1102  billets  for  each  organization  and  that  these  billets  are  sufficiently  filled  with  trained 
contracting  professionals. 

Our  analysis  of  CPAR  data  identified  some  interesting  areas  worthy  of  further 
exploration.  First,  we  would  like  to  analyze  an  expanded  database  of  services  contracts,  to 
include  other  services  in  addition  to  the  four  service  types  we  initially  researched.  With  the 
expanded  database,  we  can  analyze  overall  contract  failure  rates,  as  well  as  contract  failure 
rates  for  our  selected  contract  variables  (type  of  service,  dollar  value,  competition  level,  and 
contract  type).  Second,  we  would  like  to  further  explore  the  relationships  between 
percentage  of  filled  1102  billets  and  contract  failure  rates,  and  between  workload  dollars  per 
filled  billet  and  contract  failure  rates.  Expanding  our  contractor  performance  database  to 
include  other  service  types  will  provide  the  data  integrity  needed  to  identify  any  stronger 
relationships  among  these  contract  variables.  Finally,  we  would  like  to  analyze  the  narrative 
portion  of  the  contractor  performance  reports  to  determine  alignment  with  the  objective 
performance  ratings,  as  well  as  the  value  added,  not  only  in  the  narrative  portions,  but  also 
in  the  usefulness  of  the  CPARS  as  a  contractor  assessment  tool. 
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Ongoing  Research  Stream 


2006: 

2007: 

2008: 


2009: 

2010: 

2011: 

2012: 

2013: 


Exploratory  Research:  Opportunities  and  Challenges 

Implications  for  a  Program  Management  Approach 

Empirical  Study  of  Management  Practices  in  the  Air  Force  and 
Navy 

Empirical  Study  of  Management  Practices  in  the  Army 

Comparison  of  Acquisition  Management  Practices  in  Army, 
Navy,  and  Air  Force 

Drivers  of  Acquisition  Management  Practices  in  the  Army 

Defining  and  Measuring  Success  of  Services  Contracts  in  the 
Navy 

Drivers  of  Success  in  Services  Acquisition 
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Current  Research  Design 


Purpose  of  our  research  is  to  use  CPAR  data  to 
identify  variables  in  the  services  contracting 
process  that  drive  the  success  of  services 
acquisition. 

Research  Question:  Do  the  following  variables 
affect  the  success  of  a  service  contract? 

-  Service  Type 

-  Contract  Amount 

-  Level  of  Competition 

-  Contract  Type  4 
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Contract  Variable 


Type  of  Service 


Contract  Amount 


Level  of 
Competition 


Contract  Type 


Current  Research  Design 


CPAR  Area 


Small  Business 
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CPARS 


Contractor  Performance  Assessment  Reporting  System 


Assessment  Areas 

Quality 

Schedule 

Cost  Control 

Business  Relations 

Management  of  Key  Personnel 

Utilization  of  Small  Business 


Objective  Ratings 

Exceptional 

(5) 

Very  Good 

(4) 

Satisfactory 

(3) 

Marginal 

(2) 

Unsatisfactory 

(1) 

A  contract  for  which  the  contractor  received  either 
Marginal  or  Unsatisfactory  rating  in  any  assessment  area 
was  considered  as  an  unsuccessful  contract. 


WWW.NPS.EDU 


6 


rrr. 


NAVAL 

2^  POSTGRADUATE 
W  SCHOOL 


Empirical  Study 


Total  Contracts 

Total  Army  MICC  Non- 
System  Contracts 

14,395 

Less:  Non-R.  J,  S,  D 

Service  Contracts 

8,774 

Total  R,  J,  S,  D  Service 
Contracts 

5,621 

Less:  R,  J,  S,  D  Service 
Contracts  at  other  MICC 

4,906 

R,  J,  S,  D  Service 

Contracts  at  MICC  FDO 
Eustis,  Knox,  Hood, 

Bragg,  Sam  Houston 

715 

Fort  Eustis 

Fort  Knox 

Fort  Hood 

Fort  Bragg 

Fort  Sam  Houston 

238 

119 

114 

55 

189 
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Failures 

Success 

Total 

Failure  Rate 

Contracts 

22 

693 

715 

3.08% 

Successes 
Average  Ratings 

Failures  Average 
Ratings 

Quality 

4.19 

2.5 

Schedule 

4.19 

2.5 

Cost  Control 

4.1 

2.31 

Business  Relations 

4.17 

3 

Management  of  Key  Personnel 

4.18 

2.68 

Utilization  of  Small  Business 

4.07 

2.5 
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Empirical  Study 


Finding  1:  The  S  type  services  (Utilities  and  Housekeeping) 
had  the  highest  failure  rate  of  all  the  product  service  codes 
analyzed.  (p-v alue  =  0.761;  Not  significant) 


Type  of  Service  Acquired 
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1  mill 


Empirical  Study 


Finding  2:  Contracts  with  a  dollar  value  from  $50  million  to 
$1  billion  had  the  highest  failure  rate  of  all  the  contract 
categories.  (p-v alue  =  0.036;  Significant) 


t/i 

cu 

■M 

CO 

ca 

a > 

_3 

CO 


10.00% 
9.00 % 
8.00% 
7.00% 
6.00% 
5.00% 
4.00% 
3.00% 
2.00% 
1.00% 
0.00% 


8.70% 


2.86% 


2.15% 


2.54% 


0.00% 


$0-$lM 


>$1M-$10M  >$10M-$50M  >$50M-$1B 
Contract  Dollar  Amount  Catagories 


>$1B 


10 


NAVAL 

POSTGRADUATE 

SCHOOL 


Empirical  Study 


Finding  3:  Contracts  awarded  competitively  had  the 
highest  failure  rate  when  compared  to  the  other  contracts. 

{p-v alue  =  1 .00;  Not  significant) 


Basis  of  Contract  Award 
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Empirical  Study 


Finding  4:  Contracts  structured  as  a  combination  contract 
had  the  highest  failure  rate  when  compared  to  the  other  five 
types  of  available  contracts. 

(p-value  =  0.009;  Significant) 
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Empirical  Study 


Finding  5:  Regional  MICC  offices  that  had  spent  between  $0 
and  $500  million  in  annual  workload  had  the  highest  failure 
rate. 
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Empirical  Study 


Finding  6:  MICCs  that  completed  3,501  to  7,000  contract 
actions  annually  had  the  highest  failure  rate  when  compared  to 
MICCs  that  completed  3,500  or  fewer  contract  actions. 
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Empirical  Study 


Finding  7:  The  category  with  0  to  50  1102  billets  had  the 
highest  failure  rate. 
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Empirical  Study 


Finding  8:  As  the  percentage  of  1102  filled  billets  increased,  the 
contract  failure  rate  decreased. 
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Recommendations 


Place  additional  emphasis  on  procurement  planning,  source 
selection,  and  contract  administration  of  S  type  services 
(Utilities  and  Housekeeping). 

Provide  additional  review  of  higher  value  contracts  in  the 
areas  of  developing  PWSs,  cost  estimates,  budgets,  and 
service  delivery  schedules. 

Provide  additional  training  on  proposal  evaluation, 
specifically  evaluating  cost  and  technical  proposals,  as  well 
as  project  schedules. 

Provide  additional  emphasis  on  procurement  planning, 
source  selection,  and  contract  administration  of  combined 
cost  type  and  fixed  price  services  contracts. 

Ensure  sufficient  1102  billets  for  each  organization  and  that 
these  billets  are  filled  with  trained  contracting 
professionals.  17 
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DoD  Contract  Obligations  by  Appropriations  Account,  2012  &  2013 


Military  Personnel  &  Family  Housing 
(-19%  between  2012  and  2013) 

Corps  of  Engineers-Civil  Works 
(-2%  between  2012  and  2013) 

Military  Construction 
(-37%  between  2012  and  2013) 

Foreign  Military  Sales 
(-20%  between  2012  and  2013) 

RDT&E 

(-10%  between  2012  and  2013) 

Revolving  and  Management  Funds 
(-25%  between  2012  and  2013) 

Procurement 

(-9%  between  2012  and  2013) 

Operations  &  Maintenance 
(-15%  between  2012  and  2013) 

Other 

(-9%  between  2012  and  2013) 

Unlabeled 

(-12%  between  2012  and  2013) 
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Questions? 


dberteau@csis.org 


Full  data  tables  for  all  charts,  plus  additional  data  breakdowns  (by  DoD 
Component  and  by  Products/Services/R&D)  will  be  available  online  at 

http://www.csis.org/NSPIR/DoD  when  the  FY2013  Defense  Contract 
Trends  report  is  released  in  early  summer. 
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